[None][feat] Enable MM encoder cache on Qwen3.x and Gemma4 VLMs by 2ez4bz · Pull Request #16662 · NVIDIA/TensorRT-LLM

2ez4bz · 2026-07-21T06:13:03Z

Summary by CodeRabbit

New Features
- Added multimodal encoder caching support for Gemma 4, Qwen3-VL, Qwen3-VL MoE, and Qwen3.5 models.
- Improved reuse of image and video embeddings across requests, reducing repeated encoding work.
Documentation
- Expanded the supported-models documentation to include the newly supported multimodal optimization models.
Tests
- Added coverage confirming cached multimodal embeddings are reused correctly.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-07-21T06:26:13Z

📝 Walkthrough

Walkthrough

Gemma4 and Qwen multimodal model wrappers now use encoder-cache-aware embedding retrieval, expose multimodal embedding metadata, and declare cache support. Tests verify reuse across requests, while documentation lists the expanded supported model set.

Changes

Multimodal encoder cache support

Layer / File(s)	Summary
Gemma4 mixin and cache flow `tensorrt_llm/_torch/models/modeling_gemma4mm.py`, `tests/unittest/_torch/modeling/test_gemma4_multimodal.py`	Gemma4 adopts `MultimodalModelMixin`, routes encoding through `_get_or_encode_multimodal_embeddings`, exposes embedding metadata, and tests reuse of cached image embeddings.
Qwen cache-aware embedding routing `tensorrt_llm/_torch/models/modeling_qwen3vl.py`	Qwen3-VL adds cache-aware embedding selection, direct encoder invocation, embedding metadata properties, and encoder-cache activation.
Model support declarations and documentation `tensorrt_llm/_torch/models/modeling_qwen3_5.py`, `tensorrt_llm/_torch/models/modeling_qwen3vl_moe.py`, `docs/source/models/supported-models.md`	Qwen3.5 and Qwen3-VL MoE declare encoder-cache support, and the supported-models table lists the expanded optimization coverage.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#16554: Adds the encoder-cache-aware multimodal mixin and related encoder side-stream/cache integration used by this model-side adoption.

Suggested labels: api-compatible

Suggested reviewers: yechank-nvidia

Sequence Diagram(s)

sequenceDiagram
  participant Request
  participant MultimodalModel
  participant MultimodalModelMixin
  participant EncoderCache
  participant MultimodalEncoder
  Request->>MultimodalModel: image or video inputs
  MultimodalModel->>MultimodalModelMixin: _get_or_encode_multimodal_embeddings
  MultimodalModelMixin->>EncoderCache: lookup embeddings
  alt cache miss
    MultimodalModelMixin->>MultimodalModel: encode_multimodal_inputs
    MultimodalModel->>MultimodalEncoder: encode multimodal inputs
    MultimodalEncoder-->>MultimodalModelMixin: multimodal embeddings
    MultimodalModelMixin->>EncoderCache: store embeddings
  else cache hit
    EncoderCache-->>MultimodalModelMixin: cached embeddings
  end
  MultimodalModelMixin-->>MultimodalModel: embeddings

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is mostly a filled template and lacks the required issue/solution summary and test coverage details.	Add a short Description explaining the change and motivation, plus specific Test Coverage entries and any relevant checklist notes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the feature and matches the multimodal encoder-cache changes for Qwen3.x and Gemma4.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

tests/unittest/_torch/modeling/test_gemma4_multimodal.py (1)
765-776: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add production-path cache coverage for Qwen and Gemma.

Coverage is insufficient: this harness calls _get_or_encode_multimodal_embeddings() directly, so it does not exercise Gemma4 forward() or Qwen’s new _get_qwen_multimodal_embeddings() routing. Add cache-hit/miss tests to tests/unittest/_torch/modeling/test_gemma4_multimodal.py and the Qwen3-VL model test module, asserting two identical raw requests invoke the encoder once; parameterize the Qwen test across dense, MoE, and Qwen3.5 wrappers. Run the targeted tests under pytest tests/unittest/.

As per path instructions, “Keep feedback actionable: suggest concrete list file names and whether coverage is sufficient, insufficient, or needs follow-up outside the PR.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unittest/_torch/modeling/test_gemma4_multimodal.py` around lines 765 -
776, The cache test only exercises the helper directly and does not cover
production routing. Add cache miss/hit tests through
Gemma4ForConditionalGeneration.forward and Qwen’s
_get_qwen_multimodal_embeddings, asserting identical raw requests invoke the
encoder once; parameterize Qwen coverage across dense, MoE, and Qwen3.5
wrappers, and run the targeted tests under pytest tests/unittest/.
Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/source/models/supported-models.md`:
- Around line 129-138: Add Gemma4ForConditionalGeneration to the
MultimodalModelMixin optimization matrix, mark Multimodal Embeddings Cache as
Yes, and set Multimodal Encoder Side Stream according to its verified support
status.

---

Nitpick comments:
In `@tests/unittest/_torch/modeling/test_gemma4_multimodal.py`:
- Around line 765-776: The cache test only exercises the helper directly and
does not cover production routing. Add cache miss/hit tests through
Gemma4ForConditionalGeneration.forward and Qwen’s
_get_qwen_multimodal_embeddings, asserting identical raw requests invoke the
encoder once; parameterize Qwen coverage across dense, MoE, and Qwen3.5
wrappers, and run the targeted tests under pytest tests/unittest/.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2af008bf-08c0-455b-8e97-c6273f5a8e3d

📥 Commits

Reviewing files that changed from the base of the PR and between 4fb31cb and 7da572a.

📒 Files selected for processing (6)

docs/source/models/supported-models.md
tensorrt_llm/_torch/models/modeling_gemma4mm.py
tensorrt_llm/_torch/models/modeling_qwen3_5.py
tensorrt_llm/_torch/models/modeling_qwen3vl.py
tensorrt_llm/_torch/models/modeling_qwen3vl_moe.py
tests/unittest/_torch/modeling/test_gemma4_multimodal.py

Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>

2ez4bz requested review from a team as code owners July 21, 2026 06:13

2ez4bz requested review from kaiyux, laikhtewari and yechank-nvidia July 21, 2026 06:13

github-actions Bot assigned 2ez4bz Jul 21, 2026

2ez4bz force-pushed the dev-encoder-cache-more-models branch from 7da572a to f6fabcc Compare July 21, 2026 06:19

2ez4bz marked this pull request as draft July 21, 2026 06:23

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread docs/source/models/supported-models.md

2ez4bz force-pushed the dev-encoder-cache-more-models branch from f6fabcc to 993b8cd Compare July 22, 2026 20:21

2ez4bz added 2 commits July 23, 2026 10:01

[None][feat] Enable MM encoder cache on Qwen3.x and Gemma4 VLMs

63ca9aa

Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>

trim fat

99db94e

Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com>

2ez4bz force-pushed the dev-encoder-cache-more-models branch from 993b8cd to 99db94e Compare July 23, 2026 23:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][feat] Enable MM encoder cache on Qwen3.x and Gemma4 VLMs#16662

[None][feat] Enable MM encoder cache on Qwen3.x and Gemma4 VLMs#16662
2ez4bz wants to merge 2 commits into
NVIDIA:mainfrom
2ez4bz:dev-encoder-cache-more-models

2ez4bz commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 21, 2026

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

2ez4bz commented Jul 21, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Jul 21, 2026

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

2ez4bz commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading